This invention relates to data transmission over a serial path and methods and apparatus for increasing data throughput over the path.
The problem of transferring data from a source to a receiver recurs continuously in data processing systems. For example, such data transfer occurs both over networks on which clients and servers exchange data and over internal computer busses on which CPUs, memories and peripheral units exchange data.
Nodes coupled to a network typically communicate by exchanging messages which comprise discrete packets or xe2x80x9ccellsxe2x80x9d of data. These packets can be transported over the network with one of a variety of transport techniques. For example, asynchronous transfer mode (ATM) is a relatively new data transport technique in which the network transports and switches short, fixed-length cells. In applications utilizing ATM, data packets to be transported are first broken up into ATM cells, transmitted serially over a data link, and then reassembled at a destination. The header of an ATM cell contains the information used to transport the cell from one node to the next over a switched route which has been previously established by means of a separate signaling channel.
Input/output (I/O) subsystems allow CPUs, memories and peripheral units exchange data. In many conventional computer designs, these I/O subsystems use a shared memory mapped model with a shared bus topology. However, there are numerous shortcomings in this technology which limit the computer performance. For example, in a memory mapped bus architecture, the CPU must slow down to the speed of the bus each time communication with one of the controller cards is necessary. In addition, scalability is another prime concern and bus performance does not scale well in shared bus systems. Once all the slots on a bus are occupied, additional peripheral expansion becomes difficult. Further, devices attached to the bus must arbitrate for bus access and contention between devices for bus access can degrade the performance of all devices. System configuration is a difficult because system performance can depend on the exact order of the cards on the bus and the bus must physically be very close to the memory controller. Finally, power and cooling requirements of slot based buses must be designed to handle a fully populated system and thus the cost is increased for systems which are not fully populated.
Consequently, new internal bus technologies are being considered which overcome some of the shortcomings of the shared bus technology. One of these newer bus technologies, called xe2x80x9cNext Generation I/Oxe2x80x9d (NGIO) architecture is a channel oriented, switched point-to-point serial link architecture. The NGIO architecture uses a plurality of xe2x80x9clinksxe2x80x9d or physical connections to connect each I/O controller to a switch. The switch can then connect the I/O controllers to a specialized DMA engine called a xe2x80x9cchannel adapterxe2x80x9d which transfers data between the I/O controllers and the memory. In order to provide extensibility, switches can also be connected to other switches by means of links.
In order to move data packets over a link, specialized hardware breaks each data packet into cells and the cells are transmitted serially from a data source to a data receiver over the link. At the data receiver, the cells are reassembled by other hardware into the complete data packet.
Even though this new architecture is expected to significantly increase data throughput between the memory and I/O controllers, data flow is still limited by the serial data rate of a link and there are instances where the throughput of a single link is insufficient. A similar problem occurs in conventional switched link networks. In the NGIO system, a proposed variation called the Multi Link Extension (MLX) permits several serial links can be dynamically associated into a set called a xe2x80x9cbundlexe2x80x9d which is used to transmit cells in parallel. The bundle exhibits overall increased bandwidth and can be used to match bandwidths between the I/O controllers and the point-to-point switch. However, no mechanism for implementing bundles has been described. Consequently, there is a need for further increasing the throughput of switched link architectures.
In accordance with one illustrative embodiment of the invention, the links in a bundle, or set of related links, are ordered. A data sender sequentially transmits cells in the data packet over links in the bundle. For example, a first cell is transmitted over a first ordered link in the bundle. The next cell is transmitted over the next link in the bundle order and operation continues in a xe2x80x9cround robinxe2x80x9d fashion. The data receiver is informed in advance of the order of links in the bundle and expects cells to be distributed over the links in that order.
In accordance with one embodiment, the data sender transmits a bundle sequence number with each data cell. As each new data cell is transmitted, the bundle sequence number is incremented. The receiver uses the bundle sequence number to validate cell order, to detect lost cells, and to resynchronize cell order with the sender in the case of cell errors.
In accordance with another embodiment, bundle sequence numbers are used on a single bundle on a single link, or xe2x80x9chopxe2x80x9d, between a data sender and a data receiver. When the cells are received by the data receiver of a hop, the received bundle sequence numbers are replaced by new bundle sequence numbers generated by the data sender for transmission over the next hop.
In accordance with a further embodiment, first in, first out (FIFO) buffers are used at the data receiver in order to compensate for unequal time delays in the links in a bundle.